Json type #214

oldPadavan · 2018-06-29T14:39:52Z

Implement #195

codecov · 2018-06-29T14:57:24Z

Codecov Report

Merging #214 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #214   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          11     11           
  Lines        1327   1365   +38     
  Branches      245    249    +4     
=====================================
+ Hits         1327   1365   +38

samuelcolvin · 2018-06-29T15:27:54Z

pydantic/validators.py

+
+
+class Json(Generic[JsonObj]):  # defined not in types.py to avoid circular imports
+    pass


Move this to types.py and implement get_validators

I also don't htink you need JsonObj = TypeVar('JsonObj', Dict[str, Any], List) etc.

I've implemented JsonObj = TypeVar('JsonObj', Dict[str, Any], List) in order to be possible to write Json[List[int]], without it I was getting exceptions like JsonObj = TypeVar('JsonObj', Dict[str, Any], List) or TypeError: typing.Union[typing.List, typing.Dict] is not a generic class. So I implemented Json class a a subclass of typing.Generic. If You know other way, please let me know, because I didn't find it

samuelcolvin · 2018-06-29T15:30:23Z

pydantic/fields.py

@@ -181,6 +184,9 @@ def _populate_sub_fields(self):
        elif issubclass(origin, Set):
            self.type_ = self.type_.__args__[0]
            self.shape = Shape.SET
+        elif issubclass(origin, Json):


I don't think you need to make any changes in fields.py

The reason behind changes in fields.py is that Json[List] had __origin__ attribute, thus in _populate_sub_fields I was getting AssertionError after origin was compared against List, Dict etc.

samuelcolvin · 2018-06-29T15:30:51Z

pydantic/errors.py

+
+
+class JsonError(PydanticValueError):
+    msg_template = "{json_str} is not valid JSON and can't be decoded"


I don't think we should include the original string in the error message, it could be very long.

samuelcolvin · 2018-07-06T13:08:11Z

Ok, sounds like I need to look into it more and getting a better understanding.

samuelcolvin · 2018-07-07T15:12:40Z

ok, I would use something like

class JsonMeta(type):
    def __getitem__(self, t):
        return type('JsonWrapperValue', (JsonWrapper,), {'inner_type': t})


class Json(metaclass=JsonMeta):

    @classmethod
    def get_validators(cls):
        yield str_validator
        yield json_validator

class JsonWrapper:
    __slots__ = 'inner_type',

Naked use of Json eg.

class Foobar(BaseModel):
    x: Json

won't need anything special.

Parameterised use of Json eg.

class Foobar(BaseModel):
    x: Json[List[int]]

will need some changes to fields.py, basically you'll just need

self.type_ = self.type_.inner_type
self.parse_json = True

Then parse json before continuing with normal validation.

oldPadavan · 2018-07-10T08:05:17Z

I don't really get why deploy/netlify fails. Log says it failed to clone repository:

10:55:36 AM: git clone [email protected]:samuelcolvin/pydantic
10:56:10 AM: Error cloning repository: [email protected]:samuelcolvin/pydantic
10:56:10 AM: Failing build: Failed to prepare repo
10:56:10 AM: failed during stage 'preparing repo': exit status 128

samuelcolvin

Looking good, just a bunch of small things to fix.

Netlify was just problems with their service, I restarted the build and it's working now.

samuelcolvin · 2018-07-10T10:25:01Z

pydantic/errors.py

+
+
+class JsonError(PydanticValueError):
+    msg_template = "Not valid JSON provided - it can't be decoded"


Just Invalid JSON would be more succinct and clear.

samuelcolvin · 2018-07-10T10:26:46Z

pydantic/fields.py

-            loc = (loc,)
+        loc = loc if isinstance(loc, tuple) else (loc, )
+
+        v, error = self._validate_json(v, loc)


Please can you move the if self.parse_json to here.

so

if self.parse_json: v, error = self._validate_json(v, loc) if error: ...

I moved checking of self.parse_json to self._validate_json in order to skip from flake8 warning pydantic/fields.py:244:5: C901 'Field.validate' is too complex (11)

Thought that might be the case, just add # noqa: C901 (ignore complexity) to the end of the function definition.

samuelcolvin · 2018-07-10T10:27:22Z

pydantic/fields.py

+        if self.parse_json:
+            try:
+                return json.loads(v), None
+            except (json.JSONDecodeError, TypeError):


you can use ValueError instead of son.JSONDecodeError. Do we actually need TypeError?

I included TypeError in pydantic/fields.py to handle these cases:

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.6/json/__init__.py", line 348, in loads 'not {!r}'.format(s.__class__.__name__)) TypeError: the JSON object must be str, bytes or bytearray, not 'int'```

ok, but that should raise a descendent if TypeError not ValueError, best to use a separate except

samuelcolvin · 2018-07-10T10:42:22Z

pydantic/validators.py

@@ -239,6 +251,7 @@ def arbitrary_type_validator(v) -> type_:
    (time, [parse_time]),
    (timedelta, [parse_duration]),

+    (JsonWrapper, [str_validator, json_validator]),


This doesn't make any sense, can just be removed.

samuelcolvin · 2018-07-10T10:42:43Z

pydantic/validators.py

@@ -13,6 +14,10 @@
 NoneType = type(None)


+class JsonWrapper:


this can be moved into types.py

samuelcolvin · 2018-07-10T10:43:11Z

pydantic/validators.py

@@ -213,6 +218,13 @@ def path_exists_validator(v) -> Path:
    return v


+def json_validator(v: str):


clearer if this is moved into a validate class method on Json

samuelcolvin · 2018-07-10T10:44:19Z

pydantic/fields.py

@@ -154,11 +158,14 @@ def schema(self, by_alias=True):

    def _populate_sub_fields(self):
        # typing interface is horrible, we have to do some ugly checks
+        if inspect.isclass(self.type_) and issubclass(self.type_, JsonWrapper):


just use isinstance(self.type_, type) and issubclass(self.type_, JsonWrapper)

samuelcolvin · 2018-07-10T10:45:45Z

tests/test_types.py

+    assert JsonModel(json_obj=json.dumps(obj)).dict() == {'json_obj': obj}
+
+
+class JsonDetailedModel(BaseModel):


can you move the class definition into each test since all the field setup is done here and can fail.

samuelcolvin · 2018-07-10T10:46:19Z

tests/test_types.py

+    class JsonModel(BaseModel):
+        json_obj: Json
+
+    obj = {'a': 1, 'b': [2, 3]}


confused to dump then load. Better to use raw JSON strings in the tests

'{"a": 1, "b": [2, 3]}'

samuelcolvin

Can you also add a new line to HISTORY.rst and fix coverage.

samuelcolvin · 2018-07-10T13:11:27Z

pydantic/types.py

+    def validate(cls, v: str):
+        try:
+            return json.loads(v)
+        except json.JSONDecodeError:


use ValueError

samuelcolvin · 2018-07-10T13:11:45Z

pydantic/fields.py

-        return v, None
+        try:
+            return Json.validate(v), None
+        except (ValueError, TypeError) as exc:


split ValueError and TypeError as below.

samuelcolvin · 2018-07-10T13:14:07Z

pydantic/errors.py

+
+
+class JsonNotStrError(PydanticTypeError):
+    code = 'json_not_str'


name JsonTypeError and change code to just 'json' which will result in the final code being type_error.json

samuelcolvin · 2018-07-10T13:14:24Z

pydantic/errors.py

+
+class JsonNotStrError(PydanticTypeError):
+    code = 'json_not_str'
+    msg_template = "JSON object must be str, bytes or bytearray"


single quotes for strings please, and above.

also can you add a test for pssing in bytes.

samuelcolvin · 2018-07-10T14:39:05Z

awesome, thank you.

We also need to add docs for this. Would you be able to work on that in a separate PR? Otherwise, I'll do it when I get a chance.

oldPadavan · 2018-07-11T11:34:50Z

Should I add it to Exotic Types section or somewhere else?

samuelcolvin · 2018-07-11T11:35:36Z

I would say best if it was a new section after Exotic Types, or perhaps a new subsection of exotic types.

oldPadavan added 3 commits June 29, 2018 17:37

Json type

2fdcdf1

Sort imports

3d3b5bb

one more unsorted import

bcb5638

samuelcolvin requested changes Jun 29, 2018

View reviewed changes

oldPadavan added 3 commits July 9, 2018 16:34

Merge branch 'master' into json-data-type-195

dff5cdf

Changed the way Json type is defined

f93fbb5

Fix Function is too complex (C901) warning

6117e9d

samuelcolvin reviewed Jul 10, 2018

View reviewed changes

Fixed remarks

6f5f6be

samuelcolvin reviewed Jul 10, 2018

View reviewed changes

oldPadavan added 2 commits July 10, 2018 17:20

Fix additional remarks

0b3f2d6

Additional test

fa6edac

samuelcolvin merged commit c31b8d6 into pydantic:master Jul 10, 2018

oldPadavan mentioned this pull request Jul 11, 2018

Docs for json type #228

Merged



		class Json(Generic[JsonObj]): # defined not in types.py to avoid circular imports
		pass



		class JsonError(PydanticValueError):
		msg_template = "{json_str} is not valid JSON and can't be decoded"



		class JsonError(PydanticValueError):
		msg_template = "Not valid JSON provided - it can't be decoded"

		@@ -213,6 +218,13 @@ def path_exists_validator(v) -> Path:
		return v


		def json_validator(v: str):

		assert JsonModel(json_obj=json.dumps(obj)).dict() == {'json_obj': obj}


		class JsonDetailedModel(BaseModel):



		class JsonNotStrError(PydanticTypeError):
		code = 'json_not_str'

Uh oh!

Json type #214

Json type #214

Uh oh!

Conversation

oldPadavan commented Jun 29, 2018

Uh oh!

codecov bot commented Jun 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelcolvin Jun 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelcolvin commented Jul 6, 2018

Uh oh!

samuelcolvin commented Jul 7, 2018

Uh oh!

oldPadavan commented Jul 10, 2018

Uh oh!

samuelcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

samuelcolvin Jul 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelcolvin commented Jul 10, 2018

codecov bot commented Jun 29, 2018 •

edited

Loading

samuelcolvin Jun 29, 2018 •

edited

Loading

samuelcolvin Jul 10, 2018 •

edited

Loading